INVARIANT DISTRIBUTIONS ON NON-DISTINGUISHED 
NILPOTENT ORBITS WITH APPLICATION TO THE GELFAND 
PROPERTY OF (G'L2„(M), SpsnW) 

AVRAHAM AIZENBUD AND EITAN SAYAG 

Abstract. We study invariant distributions on the tangent space to a symmetric space. 
We prove that an invariant distribution with the property that both its support and the 
support of its Fourier transform are contained in the set of non-distinguished nilpotent 
orbits, must vanish. We deduce, using recent developments in the theory of invari- 
ant distributions on symmetric spaces that the symmetric pair {GL2n{^), Sp2n(^)) is a 
Gelfand pair. More precisely, we show that for any irreducible smooth admissible Frechet 
representation (tt, i5) of GL2„(M) the space of continuous functionals Homgp2^(^]^^{E,C) 
is at most one dimensional. Such a result was previously proven for p-adic fields in |HRj 
for C in [S]. 
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1. Introduction 

Let (V, uj) be a symplectic vector space over M. Consider the standard imbedding 
Sp(V,uj) C GL(y) and the natural action of Sp(y,uj) x Sp(y,uj) on GL(y). In this 
paper we prove the following theorem: 

Theorem A. Any Sp{V,uj) x Sp{V,uj) - invariant distribution on GL{V) is invariant 
with respect to transposition. 

It has the following corollary in representation theory: 

Theorem B. Let {V, uj) be a symplectic vector space and let E be an irreducible admissible 
smooth Frechet representation of GL{V). Then 

dimHomsp(v){E,'C) < 1 

Theorem B is deduced from Theorem A using the Gelfand-Kazhdan method (adapted 
to the archimedean case in [AGS]). 

The analogue of Theorem A and Theorem B for non-archimedean fields were proven in 
[HRj using the method of Gelfand and Kazhdan. A simple argument over finite fields is 
explained in |GGj and using this a simpler proof of the non-archimedean case was written 
in |OSj . Recently, one of us, using the ideas of [AG 2] extended the result to the case 
F = C(see[S]). 

Our proof of Theorem A is based on the methods of |AG2j . In that work the notion of 
regular symmetric pair was introduced and shown to be a useful tool in the verification of 
the Gelfand property . Thus, the main result of the present work is the regularity of the 
symmetric pair {GL{V), Sp{V,uj). In previous works the proof of regularity of symmetric 
pairs was based either on some simple considerations or on a criterion that requires nega- 
tivity of certain eigenvalues (this was implicit in [JRj . |RRj and was explicated in |AG2j . 

UGH], non, m- 

The pair {GL{V), Sp{V, uj)) does not satisfy the above mentioned criterion and requires 
new techniques. 

1.1. Main ingredients of the proof. 

To show regularity we study distributions on the space g'^ = {X G ghn '■ JX = XJ} 
where ~ ^ j*^ ^o"^ ' ^^^^ precisely, we are interested in those distributions that 

are invariant with respect to the conjugation action of Sp2n and supported on the nilpotent 
cone. To classify the nilpotent orbits of the action we use the method of [GG] to identify 
these orbits with nilpotent orbits of the adjoint action of GLn on its Lie algebra. This 
allows us to show that there exists a unique distinguished nilpotent orbit O and that this 
orbit is open in the nilpotent cone. Next, we use the theory of D modules, as in |AG5j . to 
prove that there are no distributions supported on non-distinguished orbits whose Fourier 
transform is also supported on non-distinguished orbits (see Theorem 13.0. lip . 

1.2. Structure of the paper. 

In section |2] we give some preliminaries on distributions, symmetric pairs and Gelfand 
pairs. We introduce the notion of regular symmetric pairs and show that Theorem 7.4.5 
of |AG2j and the results of [S] allow us to reduce the Gelfand property of the pair in 
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question to proving that the pair is regular. In section [3] we prove the main technical 
result on distributions, Theorem 13.0. Ill It states that under certain conditions there are 
no distributions supported on non-distinguished nilpotent orbits. The proof is based on 
the theory of D-modules. In section H] we use Theorem 13.0.111 to prove that the pair 
{GL{V), Sp{V,uj)) is regular. 

1.3. Acknowledgements. 

We thank Dmitry Gourevitch and Omer Offen for fruitful discussions. Part of the work on 
this paper was done while the authors visited the Max Planck Institute for Mathematics in 
Bonn. The visit of the first named author was funded by the Bonn International Graduate 
School. The visit of the second named author was partially funded by the Landau center 
of the Hebrew University. 

2. Preliminaries 
2.1. Notations on invariant distributions. 

2.1.1. Schwartz distributions on Nash manifolds. 

We will use the theory of Schwartz functions and distributions as developed in |AG1] . 
This theory is developed for Nash manifolds. Nash manifolds are smooth semi-algebraic 
manifolds but in the present work only smooth real algebraic manifolds are considered. 
Therefore the reader can safely replace the word Nash by smooth real algebraic. 

Schwartz functions are functions that decay, together with all their derivatives, faster 
than any polynomial. On it is the usual notion of Schwartz function. For precise 
definitions of those notions we refer the reader to |AGlj . We will use the following 
notations. 

Notation 2.1.1. Let X be a Nash manifold. Denote by S{X) the Frechet space of 
Schwartz functions on X . 

Denote by S*{X) := S{X)* the space of Schwartz distributions on X. 

For any Nash vector bundle E over X we denote by S{X,E) the space of Schwartz 
sections of E and by S*{X,E) its dual space. 

Notation 2.1.2. Let X be a smooth manifold and let Z G X be a closed subset. We 
denote := G 5*(A)|Supp(^) C Z}. 

For a locally closed subset Y C X we denote SxiY) '■= '^x\(y\y)^^'^' same way, 

for any bundle E on X we define Sx{Y, E). 

Remark 2.1.3. Schwartz distributions have the following two advantages over general 
distributions: 

(i) For a Nash manifold X and an open Nash submanifold U G X , we have the following 
exact sequence 

^ Sx{X \U) ^ S*{X) S*{U) 0. 
(a) Fourier transform defines an isomorphism T : 5*(M") 
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2.1.2. Basic tools. 

We present here some basic tools on equivariant distributions that we will use in this 
paper. 

Proposition 2.1.4. Let a Nash group G act on a Nash manifold X. Let Z d X he a 

closed subset. 

Let Z = Ul^^Zi be a Nash G-invariant stratification of Z. Let x be a character of G. 
Suppose that for any k G Z>o and < i < I we have S*{Zi, Sym^ {C N^'))'^'^ = 0. Then 
S*x{Zf'>' = 0. 

This proposition immediately follows from Corollary 7.2.6 in |AGSj . 

Theorem 2.1.5 (Frobenius reciprocity). Let a Nash group G act transitively on a Nash 
manifold Z. Let (f : X ^ Z be a G-equivariant Nash map. Let z & Z. Let G^ be 
its stabilizer. Let X^ be the fiber of z. Let x be a character of G. Then S*{X)'^'^ is 
canonically isomorphic to 5*(X2)'^^''^'^°''^^'^Gz where A denotes the modular character. 

For proof see |AG2j . Theorem 2.3.8. 

2.1.3. Fourier transform. 

From now till the end of the paper we fix an additive character k of M given by k{x) : = 

Notation 2.1.6. Let V be a vector space over R. Let B be a non- degenerate bilinear 
form on V . Then B defines Fourier transform with respect to the self-dual Haar measure 
onV. We denote it by Tb ■ S*{V) S*{V). 

For any Nash manifold M we also denote by J^b '■ S*{M x V) — > S*{M x V) the partial 
Fourier transform. 

If there is no ambiguity, we will write Ty , and sometimes just T , instead of Tb- 

We will use the following trivial observation. 

Lemma 2.1.7. Let V be a finite dimensional vector space over M. Let a Nash group G 
act linearly on V. Let B be a G-invariant non- degenerate symmetric bilinear form on V . 
Let M be a Nash manifold with an action of G. Let ^ G iS*(V(]R) x M) be a G-invariant 
distribution. Then Tb{C) ^■5 o/so G-invariant. 

2.2. Gelfand pairs and invariant distributions. 

In this section we recall a technique due to Gelfand and Kazhdan (see |GK] ) which allows 
to deduce statements in representation theory from statements on invariant distributions. 
For more detailed description see [AGSj . section 2. 

Definition 2.2.1. Let G be a reductive group. By an admissible representation of 

G we mean an admissible smooth Frechet representation of G(M). 

We now introduce three notions of Gelfand pair. 

Definition 2.2.2. Let H G G be a pair of reductive groups. 

• We say that {G,H) satisfy GPl if for any irreducible admissible smooth Frechet 
representation (vr, E) of G we have 

dim H am H(^){E,C) < 1 
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• We say that {G,H) satisfy GP2 if for any irreducible admissible smooth Frechet 
representation (vr, E) of G we have 

dim H am H(R){E,C) ■ dim Ham h{r){E,C) < 1 

• We say that {G, H) satisfy GPS if for any irreducible unitary representation 
(tt, TC) of G{R) on a Hilbert space H we have 

dimHomH(R){n°°X) < 1- 

Property GPl was established by Gelfand and Kazlidan in certain p-adic cases (see 
|GK] ). Property GP2 was introduced in |Groj in the p-adic setting. Property GPS was 
studied extensively by various authors under the name generalized Gelfand pair both 
in the real and p-adic settings (see e.g. |vDl IBvDj ). 

We have the following straightforward proposition. 

Proposition 2.2.3. GPl GP2 GP3. 

We will use the following theorem from [AGSj which is a version of a classical theorem 
of Gelfand and Kazhdan. 

Theorem 2.2.4. Let H G G be reductive groups and let r be an involutive anti- 
automorphism of G and assume that t{H) = H . Suppose r(^) = ^ for all bi H{M)- 
invariant distributions ^ on G{M.). Then {G,H) satisfies GP2. 

In our case GP2 is equivalent to GPl by the following proposition. 

Proposition 2.2.5. Suppose H C GL„ is transpose invariant subgroup. Then GPl is 
equivalent to GP2 for the pair (GL„, H). 

For proof see |AGSj . proposition 2.4.1. 

Corollary 2.2.6. Theorem A implies Theorem B. 

2.3. Symmetric pairs. 

In this subsection we review some tools developed in [AG 2] that enable to prove that, 
granting certain hypothesis, a symmetric pair is a Gelfand pair. 

Definition 2.3.1. A symmetric pair is a triple {G,H,6) where H G G are reductive 
groups, and 9 is an involution of G such that H = G^ . In cases when there is no ambiguity 
we will omit 9 

For a symmetric pair {G,H,9) we define an anti-involution a : G —>■ G by cr{g) : = 
9(g~^), denote q := LieG, f) := LieH , 0°" := {a E Q\9{a) = —a}. Note that H acts on q"' 
by the adjoint action. Denote also G" := {g G G\a{g) = g} and define a symmetriza- 
tion map s : G(R) G^iR) by s{g) := g(y{g). 

The following lemma is standard: 

Lemma 2.3.2. The symmetrization map s : G ^ G" is submersive and hence open. 

Definition 2.3.3. Let {Gi, Hi,9i) and (6*2,^/2,^2) be symmetric pairs. We define their 
product to be the symmetric pair {Gi x G2,Hi x H2,9i x 92). 

Definition 2.3.4. We call a symmetric pair {G, H, 9) good if for any closed H(M.) x H(M.) 
orbit O G G(R), we have (r{0) = O. 
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Definition 2.3.5. We say that a symmetric pair {G, H, 9) is a GK pair if any H(M.) x 
H(R) - invariant distribution on G'(M) is a - invariant. 

Definition 2.3.6. We define an involution 9 : GL2n GL2n by 9{x) = Jx*J^^ where 
J = ^ ^^"^ ■ ^'^^^ ^^^^ {GL2n, Sp2n, 9) is a symmetric pair. 

Theorem A can be rephrased in the following way: 
Theorem A'. The pair {GL2n, Sp2n) defined over M zs a GK pair. 

2.3.1. Descendants of symmetric pairs. 

Proposition 2.3.7. Let {G,H,9) be a symmetric pair. Let g G G{M.) such that HgH is 
closed. Let x = s{g). Then x is semisimple. 

For proof see e.g. [AG 2] . Proposition 7.2.1. 

Definition 2.3.8. In the notations of the previous proposition we will say that the pair 
{Gx, Hx,9\g^) is a descendant of {G,H,9). Here Gx (and similarly for H) denotes the 
stabilizer of x in G. 

2.3.2. Regular symmetric pairs. 

Notation 2.3.9. Let V be an algebraic finite dimensional representation over M o/ a 
reductive group G. Denote Q{V) := V/V'^. Since G is reductive, there is a canonical 
embedding QiV) * V . 

Notation 2.3.10. Let {G,H,9) be a symmetric pair. We denote by Mch the subset of 
all the nilpotent elements in Q{q^). Denote Rg,h '■= Qi^") — -^ch- 

Our notion of Rg,h coincides with the notion R{q") used in |AG2j . Notation 2.1.10. 
This follows from Lemma 7.1.11 in |AG2j . 

Definition 2.3.11. Let vr be an action of a real reductive group G on a smooth affine 
variety X . We say that an algebraic automorphism t of X is G -admissible if 
(i) 7r(G'(]R)) is of index at most 2 in the group of automorphisms of X generated by 
7r(G(M)) and r. 

(a) For any closed G{R) orbit O C X{R), we have t{0) = O. 

Definition 2.3.12. Let {G,H,9) be a symmetric pair. We call an element g G G(]R) 
admissible if 

(i) Ad{g) commutes with 9 (or, equivalently, s{g) G Z{G)) and 
(a) Ad{g)\giT is H -admissible. 

Definition 2.3.13. We call a symmetric pair {G,H,9) regular if for any admissible 
g G G(M) such that every H{M.) -invariant distribution on Rg,h is also Ad{g) -invariant, 
we have 

(*) every H (M.) -invariant distribution on Q{q'^) is also Ad{g) -invariant. 

Clearly, the product of regular pairs is regular (see [AG 2] . Proposition 7.4.4). 
We will deduce Theorem A' (and hence Theorem A) from the following Theorem: 

Theorem C. The pair {GL2n, Sp2n) defined over M is regular. 
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The deduction is based on the following theorem (see [AG 2] . Theorem 7.4.5.): 

Theorem 2.3.14. Let {G,H,6) be a good symmetric pair such that all its descendants 
(including itself) are regular. Then it is a GK pair. 

Corollary 2.3.15. Theorem C implies Theorem A. 

Proof. The pair {GL2n, Sp2n) is good by Corollary 3.1.3 of [Sj. In [S] it is shown that all 
the descendance of the pair {GL2n, Sp2n) are products of pairs of the form {GL2m, Sp2m) 
and {{GL2m)c/M., {Sp2m)c/R)y here Gc/r denotes the restriction of scalars (in particular 
Gc/r(IR) = G{C)). By Corollary 3.3.1. of the pair {{GL2m)c/R, {Sp2m)c/R) is regular. 
Now clearly Theorem C implies Theorem A' and hence Theorem A. □ 

We denote by 0(F) the space of regular functions on the algebraic variety V. We will 
also need the following Proposition, which must be well known. 

Proposition 2.3.16. Let vr : 0°" ^ Spec{0{Q'^))^ be the projection. Let x G Mch be a 
smooth point. Then it submersive at x. 

Proof Let J = {f e Oid")^ : /(O) = 0}. By Theorem 14 of [KR|, J is a radical ideal. 
Using the Nullstellensatz, this implies that Ker^dx'n) = Tx{Ng,h)- This proves that vr is 
submersive. □ 

2.4. Singular support of distributions. 

In this subsection we introduce an important invariant SS{^) of a distribution ^ and list 
some of its properties. For more details see [AGS] . 

Notation 2.4.1. Let X be a smooth algebraic variety. Let ^ e 5*(X(R)). Let be 

the Dx submodule of S*{X{E.)) generated by ^. We denote by SS{^) C T*X the singular 
support of (for the definition see [Borj ). We will call it the singular support of^. 

Remark 2.4.2. 

(i) A similar, but not equivalent notion is sometimes called in the literature a 'wave front 
ofV- 

(a) In some of the literature, singular support of a distribution is a subset of X not to be 
confused with our SS{^) which is a subset ofT*X. We use terminology from the theory of 
D modules where the set SS{C,) is called both the characteristic variety and the singular 
support of the D module M^. 

Notation 2.4.3. Let {V, B) be a quadratic space. Let X be a smooth algebraic variety. 
Consider B as a map B : V V* . Identify T*{X x V) with T*X x V x V* . We define 
Fv : T*{X xV)-^ T*{X x V) by Fv{a,v,(p) := {a, -B-^^, Bv) . 

Definition 2.4.4. Let M be a smooth algebraic variety and uj be a symplectic form on it. 
Let Z d M be an algebraic subvariety. We call it M-co-isotropic if one of the following 
equivalent conditions holds. 

(1) The ideal sheaf of regular functions that vanish on Z is closed under Poisson 
bracket. 
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(2) At every smooth point z ^ Z we have T^Z D (T^Z)-'-. Here, {T^Z)^ denotes the 
orthogonal complement to (T^Z) in (T^M) with respect to u. 

(3) For a generic smooth point z E Z we have T^Z D (T^Z)-*-. 
// there is no ambiguity, we will call Z a co-isotropic variety. 

Note that every non-empty M-co-isotropic variety is of dimension at least ^dirriM. 

Notation 2.4.5. For a smooth algebraic variety X we always consider the standard sym- 
plectic form on T*X . Also, we denote by px '■ T*X — > X the standard projection. 

Let X be a smooth algebraic variety. Below is a list of properties of the Singular 
support. Proofs can be found in |AG5j section 2.3 and Appendix B. 



Property 2.4.6. Let ^ G 5*(X(R)). Then Supp(Ozar = Px{SS{i)){M), where 
Supp(Ozar denotes the Zariski closure o/Supp(^). 

Property 2.4.7. 

Let an algebraic group G act on X. Let q denote the Lie algebra of G. Let ^ G 
5*(X(M))^W. Then 

SSiO C {{x, 0) G T*X I Va G 0(a(x)) = 0}. 

Property 2.4.8. Let {V, B) be a quadratic space. Let Z G X x V be a closed subva- 
riety, invariant with respect to homotheties in V. Suppose that Supp(^) C 2'(M). Then 
SS{:Fv{0)^Fy{p-^\y{Z)). 

Finally, the following is a corollary of the integrability theorem ( [KKSj . [Malj . [Gab]): 

Property 2.4.9. Let X be a smooth algebraic variety. Let C, G Then SS{^) 

is co-isotropic. 

3. Invariant distributions supported on non-distinguished nilpotent 

orbits in symmetric pairs 

For this section we fix a symmetric pair (G, H, 9). 

Definition 3.0.10. We say that a nilpotent element x E is distinguished if 

Qx n g(0") c ^^G,H 

Theorem 3.0.11. Let A C J^g.h be an H invariant closed subset and assume that all 
elements of A are non- distinguished. Let W = Then W n T{W) = {0}. 

Remark 3.0.12. We believe that the methods of |SZj allow to show the same result without 
the assumption of H-invariance. 

The proof is based on the following proposition: 

Proposition 3.0.13. Let A C Mg,h be an H invariant closed subset and assume that 
all elements of A are non- distinguished. Denote by B = {(a,/?) E A x A : = 
0} C QiQ'') X QiQ"). Identify T*(q{q'')) with Qi^") x Qi^"). Then there is no non-empty 
T*{Q{Q'^))-co-isotropic subvariety of B. 
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Proof. Stratify A by finite many orbits Oi, Or- Let p : A x A A he the projection 
onto the first factor. By inductive argument it is enough to show that, for any orbit O, 
p~^{0) does not include a non empty co-isotropic subvariety. Consider the set 

Co = {ia,b):aeO,be Q(g'^), [a, b] = 0}. 

Then dim{Co) = dim{Q{Q")). Since O is not distinguished, p~^{0) is a closed subvariety 
of Co which does not include any of the irreducible components of Co- This finishes the 
proof. □ 

Proof of Theorem \nm[ Let ^ G H T{W) and let B be as in proposition |3iU31 By 
properties [2M 12X71 [2A8] we conclude that SS{C) C B. But by Property 12X91 it is 
co-isotropic and hence by Proposition 13.0.13) it is empty. Thus ^ = 0. □ 

4. Regularity 

In this section we prove the main result of the paper: 
Theorem C. The pair {GL2n, Sp2n) defined over M is regular. 

For the rest of this section we let {G, H) to be the symmetric pair (GL2n(R), Sp2n(^)- 
4.L H orbits on q'^ . 

Proposition 4.1.1. There exists a unique distinguished H-orbit in J^g,h{^)- This orbit 
is open in AfcniM) and invariant with respect to any admissible g E C- 

For the proof we will use the following Proposition (this is Proposition 2.1 of ^GGj ): 

Proposition 4.1.2. Let F be an arbitrary field. For x G GLn{F) define 

= (o /„) 

Then 7 induces a bijection between the set of conjugacy classes in GLn{F) and the set 
of orbits of Sp2n{F) x Sp2n{F) in GL2n{F). 

Corollary 4.1.3. Let d : gl„ — > q'^ be defined by 




Then d induces a bijection between nilpotent conjugacy classes in gl„ and H orbits in 
Mg,h- 

Proof. Let s : GLg^ GL"^^ be given by s{g) = ga{g)- Let W = s(G'L2„(M)). By 
Proposition 14. 1.21 the -5° 7 induces a bijection between conjugacy classes in GL„(]R) 
and H orbits on W- 

Let e ■- M ^ GLn be given by e{X) = 1 + X where M is the cone of nilpotent elements 
in gl„. Let I --W q'^ given by l{w) = w — 1. 

Then, it is easy to see that the map d\j^ : N — Mg,h coincides with the composition 
£050706. 

To finish the proof of the Proposition it is enough to show that ^{W) contains all 
nilpotent elements. Indeed, by lemma [2.3.21 the set W = s{GL2n{^)) is open and thus 
£{W) is open and hence contains all nilpotent elements. □ 
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We are now ready to prove the proposition. 



Proof of Proposition \4-l-l\ It is easy to see that if X is non regular nilpotent then 



d{X) is not distinguished. Also, a simple verification shows that if X = J„ is a 
standard Jordan block then d{Jn) is distinguished. Thus we only need to show that 
C = Ad{H)d{Jn) is open in Mq^h- For this we will show that C is dense in Mg,h- Indeed, 
C D d{Ad{GLn)Jn) = d{Af), where J\f is the set of nilpotent elements in gin. But C is 
Ad{H)-inva.nant and this implies that C = Men D 

4.2. Proof of Theorem C. 

The theorem follows from Theorem 13.0.111 and the next Proposition: 

Proposition 4.2.1. Let g E G he an admissible element. Let A he the union of all non- 
distinguished elements. Note that A is closed. Let ^ he any H invariant distrihution on 
which is anti-invariant with respect to Ad{g). Then Supp{^) C A. 

Proof. Let Oq C J\fG,H be the distinguished orbit. Let H = {Ad{H) , Ad{g)) be the group 
of automorphisms of g"^ generated by the adjoint action of H and g. Let x be the character 
of H defined by x{H — H) = — 1. We need to show 

By Proposition 12.1.41 it is enough to show 

S*iOo,Sym\CNg;^'^^)f'^ = 

Notice that H acts trivially on Spec{0{g'^))^ . Hence, by Proposition 12.3.161 the bundle 
A^g^^^"^ is trivial ClS db H bundle. This completes the proof. □ 
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